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ABSTRACT 

A quality assurance case study was conducted at the School of 
Business Studies of the University of Seville, Spain. This study explored the 
degree of satisfaction expressed by students, investigating the teaching 
performance of each university lecturer, subject program design, 
characteristics of the teaching center, issues related to the University, 
profile of the student, and the importance attached by students to each 
component of the teaching service. A total of 23,585 questionnaires were 
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responses represented 1,560 students. Findings indicate that the instrument 
demonstrated validity in measuring different levels of student satisfaction. 
Findings also indicate that the clarity of the lecturers’ explanations and 
the extent to which they are able to lighten these and make them accessible 
to students are the most significant determinants of overall student 
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This paper reports on a quality assurance case study of the University of Seville’s 
School of Business Studies. It explores the degree of satisfaction expressed by university 
students. The model uses the primary information derived from the questionnaire, 
composed of the following: the teaching performance of each university lecturer, subject 
program design, characteristics of the teaching centre, issues related to the University, 
profile of the student, and importance attached by students to each component of the 
teaching service. Findings of the case studies indicate that the tool instrument demonstrated 
itself validity in achieving the objective of measuring different levels of student expressions 
of satisfaction with the services offered. The findings suggested also that the clarity of the 
lecturers’ explanations and the extent to which they are able to lighten these and make them 
accessible to students, are most significant determinants of the overall student satisfaction. 




1. INTRODUCTION 

If learning is the home ground of the University, then the essential features that set 
it apart as an organisation, namely the results of that learning, are the key that motivates 
change, and the consequences what is taught (Allan, 1996). The mechanisms, rules and 
principles that have guided the evaluation of university teaching, and which are now 
applied to entire degree programmes, have yet to provide an adequate in-depth analysis of 
the processes involved in, and consequences of the complex task of triangulating beliefs, 
intentions and actions when we create and transmit knowledge. Evaluating teaching within 
the University is a process of filling in the gaps of content, context and perspective in a 
process of communication between the participants. It is a profound analysis of the 
architectural structure of well-organised knowledge; of professional responsibility; of the 
culture of feedback and openness; and of global relationships, whether vertical or 
horizontal. It transcends the study of discrete^ actions that can offer only a limited view of 
the mission and values established by the institution (Pratt, 1997). 

The assessment of quality in universities represents an attempt to discover the 
opportunities available through teaching, research and management, to respond to the 
prescribed institutional objective of promoting the personal development of students. Its 
purpose is to identify those elements in an apparently diverse flow of organisations, 
developments and teaching meanings that are in or out of tune for the individual student. 

The present study of student satisfaction at the University of Seville (Spain) is the 
product of a scientific need - the emergence of a wealth of research into quality assurance 
(Harman, 1998) - and of an opportunity given by the quality assessment programmes 
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welcomed and actively promoted by Spain’s Consejo de Universidades, the central 
government body that coordinates the universities. 

However, the study of student satisfaction is not simply the result of a chance 
opportunity. Student satisfaction with the teaching received year on year in a degree 
programme is the language that expresses the organizational culture of the centre where 
that programme is taught. This is the catalyst in all of the questionnaires that evaluate a 
university lecturer. As such, it acts primarily as a lyrical element that manifests the sudden 
revelation of the significance of knowledge, and of the expanded horizons of the job 
market. Through their expression of satisfaction, students show their attitudes, motivation, 
commitments, involvement, and self-discipline in their learning. Student behaviour is 
determined and made patent, and the image of the institution as a collective entity is 
projected. (Hamash-Glezer y Meyer, 1991). When a university as an organisation wants to 
demonstrate its transparency without evading the unquestionable yet invisible backcloth of 
student expectations, that institution has set out on the path towards a learning organization 
model that faces up to uncertainty and accepts it as an opportunity for change and 
transformation. It finds itself face to face with bureaucratic organisations in which the 
participants (lecturers and students) have expectations that are determined by the roles and 
functions openly structured for them through regulations and statutes with which they are in 
conflict. 

In order to study student satisfaction, as if we were studying attitudes or self- 
concept and anxiety (Townsend and others, 1998), we need to characterize the construct, 
give examples, flesh out the details, and establish a vocabulary in the text of our research 
instrument that avoids any complex academic language and, enables students to connect 
easily with their personal world. Consequently, the task of reviewing research in order to 
discover indicators of satisfaction; writing alternative versions of our test instrument; 
constructing indices of the terms and lexical items which define the concept; building a 
kind of implicit hypertext that embraces all of the references to curricular practice, and the 
cross-references to authoritative sources within the various social fields (business studies 
and education), demands highly meticulous observation. As a result, we can objectively say 
that this study gives details of the morphology of student satisfaction within the population 
of the University of Seville and describes the scope of a comprehensive research tool that 
has been empirically tested and refined. 

1. 1. Background to the study 

Quality, when it truly becomes part of the fabric of a human organization, in the 
form of experimentation and improvement, turns itself into the real text of the University 
context. Fine strands of quality interweave in a process of development that is none other 
than that of a style of learning that solves problems, learns from past experience and from 
the experience of others, and transfers knowledge quickly and efficiently throughout the 
organisation (Meade, 1995). 

Reasearch into quality in the University does not seek to account for the current 
state of affairs or to fix the institution on a certain timescale. It does not count what may be 
lost minute by minute, nor is it a narrative of abandonment that leaves the lecturer alone in 
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the face of change. Research into quality in the University is, principally, a line of thought 
about excellence, good teaching, and innovation. In a word, it is an interpretation of the 
world, a hymn to the future (Aylett y Gregory, 1996). It is no mere chance that the 
vocabulary we use to identify quality should include terms like ‘excellence’, ‘high 
standards’ or ‘markers’, or ‘achievement of objectives’. Quality in the University must be 
judged by the solidity and consistency of achievements, and also by the need to 
differentiate the services on offer (Middlehurst, 1995). 

There are enough educational and organizational reasons to support the belief that 
differences between individuals and their context interact. Moreover, it is convenient to 
distinguish between the variables involved - task, time spent studying, approaches to 
learning or academic results, among others - that explain models (Kember and others, 
1996), or to identify the meanings behind the words students use when they clamour for 
what they want, or denounce regressive teaching methods and protest about them. In the 
University context we obtain data about students’ motivations towards learning, their 
orientation towards the results of learning, their study strategies, their attitudes towards the 
subject. We are also able to compare the skills displayed by students at one centre with 
those of another (Archer and Scevak, 1998). Two of the features that these studies have in 
common are the fragmentation or discontinuous decompositon of variables, and the search 
for a mathematical theory based on models of regression. 

It was in this vein that research into teaching via the evaluation of students began in 
the early 20^'’ century (Marsh and Bailey, 1993). It may well appear as if, all in all, this line 
of research was flawed. It was centered on the instructional role of the teacher in the 
classroom, on the vast range of teaching methods, and on its well known 
multidimensionality, which meant that the stimuli selected varied greatly. This research 
was the product of daily classroom observation and of a markedly positivist culture. 

The evaluation of satisfaction represents a step forward in the process that leads to 
the full development of a learning organization. It is a research process that can blend 
together different elements, especially those that do not naturally go together. Through the 
choice of items to include in the questionnaire, we can cover aspects of teaching 
performance as well as other dimensions of quality concerned with non-educational facets 
of the institution. However, there is such a firm wish to go beyond the narrow limits of time 
that studies have been carried out which use a short term filter to look at the characteristics 
of centres and the experience of students in order to reach conclusions about the 
achievement of satisfaction in the long-term. There are studies that propose a long-term 
satisfaction for students, as if this was the motor for integration in their social and 
professional life, mixing past and present commitments, and a deep level of training (Knox, 
Lindasay and Kolb, 1992). 

Since time immemorial, a good grade has served as the essential metaphor of 
student satisfaction. The students’ position, as a means of protecting themselves in 
evaluation, may have contaminated those evaluative studies that have tried to determine 
levels of professional teaching excellence. Cases have been documented of lecturers who 
give higher grades while demanding fewer study tasks and who students perceive as 
“better” teachers (Brodie, 1998). 
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In this brief review of the research into evaluation, we refer to the different views 
that lecturers and students have about grades, and go further into the issue in order to 
demonstrate their differing perceptions. We emphasize the divergent views of the two 
groups as to the importance and impact of grades in the present and in the future, the 
proceses by which grades are determined, and the basis on which the validity of grades can 
be questioned. (Goulden y Griffin, 1997). Student evaluation can, accordingly, develop the 
protected area of motivation towards learning. Here, we are talking about a matrix of 
concepts - performance, recognition, responsibility, supervision, and status, among others - 
where certain ideas and principles become bound up together in a web of hypothetical 
relationships (Elton, 1996). 

At the moment, it seems necessary to call for reflection on the part of the lecturers, 
through which we can link evaluation with the creative freedom inherent to each subject. 
Teachers’ craft knowledge, from which they construct texts, and research into putting the 
curriculum into practice, are stimulated from the inside by reason and emotion when 
lecturers turn to their knowledge of the makeup of the class, or when they commit 
themselves to carrying out innovations in order to optimize learning on the part of their 
students (Van Driel and others, 1997). Initially, this comes about through their newly 
acquired ability to observe indicators that remain hidden from the majority. Moreover, their 
expressive evaluation capacity can lead lecturers to adopt a peer review model in order to 
further their training. Lecturers are able to rely on their personal insights into the apparently 
dead-end situations that others find themselves in, and on their intuitions about what 
actually indicates communication. This can enable them to pressure their colleagues to 
show them aspects of classroom life and to face up to the realities of their own classrooms 
in an atmosphere of mutual support. 

1.2. Paradigm of research into learning organizations 

The definition of learning encompasses the quantitative increase in knowledge, 
memorization, the acquisition of facts or methods, the abstraction of meanings, and the 
interpretative process leading to the comprehension of reality. It is one of the many 
psychological constructs featured in the textbooks, and one that recently has become a 
metaphor for the modern vision of the University as an organization (Brockbank y McGill, 
1998). The learning organization is a way of looking at processes in suspension, floating 
uninfluenced by the gravity of the short-term. It explains change in terms of statements of 
probablility or modes of behaviour visible only when brief shafts of light - indicators - 
glance off them (Slappendel, 1996). One of the essential characteristics of a learning 
organization is that it should know about the strengths required and the internal tasks 
involved in constructing the ability to learn. This can be synthesized in the idea of 
integration that unites the mission and vision in the values expressed by the organization, 
the leadership, experimentation, transfer of knowledge, teamwork and cooperation. In a 
learning organization, no one shrinks from the task of collecting information about the gap 
that exists between present actions and desired performance. On the contrary, 
organizational curiosity keeps the door wide open, and is concerned to measure the key 
factors within the institution (including student satisfaction). The learning organization 
offers its end-users the key to keep the door permanently open through new means of 
communication, including the Internet. It puts into place a modus operandi of continuing 
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education that manages change through a range of instruments; spreads sound judgements 
drawn from a range of sponsors and sources. The learning organization bears witness by the 
perseverance of a style of leadership that reveals learning initiatives, and recognises the 
systemic interdependence of the institution where the agents, their functions, and elements 
of the curriculum and materials are not diffuse but mutually interconnected. The 
profesional lives of lecturers are not locked away in files in a personal department, reduced 
to mere bureaucratic data. In the modern vision of the profesional university teacher we 
project the image of policy-maker of their personality, of their interaction with the other 
agents - lecturers and students, adminstration and service staff, and social representatives — 
to whom they have moral commitments (Evans, 1997). 

2. METHODOLOGY 

Whether they like it or not, private sector organisations are subject to market forces 
that permanently oblige them to keep their resources and their distinctive features up to 
date. They do not consider learning to be a mark of “super-excellence” but, rather, an 
essential guarantee of survival. In contrast, public institutions require leadership, vision and 
a commitment to the future, to show their concern to keep themselves facing up to the 
demands of today and of the days to come. If this leadership is absent - and, sadly, this is 
the case more often than any of us would wish - we run a serious risk of finding ourselves 
with a public sector anchored to the social demands of the past, and questioned by those of 
the present. 

The University, as the public institution that it is, cannot ignore this philosophy of 
work. In fact, we would say that it should take the lead in the entire process. A critical 
spirit, innovation, creativity, change, dynamism, and other similar characteristics have 
historically always been present throughout the process of development of the universities. 
Precisely these are the characteristics that have since ancient times earned the universities 
respect and social standing as institutions capable of generating progress. We believe that 
nowadays this view of the universities can be maintained only if they adopt a learning 
perspective which explores the future imaginatively, internalises social demands, and 
adapts their own internal processes in order to respond fully. 

The starting point for this concept of management must inevitably involve the 
creation of sophisticated mechanisms to obtain information from the setting. This must be 
fully analyzed, and transformed into managerial decisions that can immediately be 
converted into actions. This is the need that dominated our process of constructing both the 
questionnaire, on which we based the research presented here, and the model used to handle 
the information obtained from it. 

In order to measure the degree of satisfaction expressed by university students, we 
divided the questionnaire into six sections, each of which focussed on a different area of 
interest: 

1 . The teaching performance of each university lecturer (TP). 

2. Aspects of the program of each subject (Subject program design, SPD). 
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3. Characteristics of the teaching centre and of the degree programme that the 
students were following (CD). 

4. Issues related to the University beyond the scope of a specific centre (supra- 
centre services, SCS). 

5. Questions related to establishing the profile of the student respondent. 

6. The relative importance attached by students to each component of the 
teaching service, as part of obtaining an overall quality service. 

We selected the questions to include in each module out by transforming 
recommendations and proposals made by the European Foundation for Quality 
Management (EFQM) in order to adapt these to the specific characteristics of the higher 
educational institution and to the information available to the interviewees (EFQM, 1995). 
Finally, each question was cast affirmatively in order to measure it against a five point 
Likert scale. 

The data was processed using the model described earlier, which we constructed in 
order to achieve three objectives: 

a) Easy to apply. Throughout, we aimed to apply the model in such a way as to 
avoid an excessively complicated mathematical process. We wanted to ensure that 
anyone interested in using it would not by put off by the operational aspects. 

b) Measurement and identification of areas for improvement. In line with TQM 
philosophy, quality management implies introducing into the organization 
systematic processes to measure the levels of excellence attained, and to identify 
those aspects of activities that need reformulation and improvement. Based on this 
belief, the model enabled us to obtain different indicators of quality of service 
(always seen from the viewpoint of student satisfaction and supported by an 
analysis of the raw data), at the same time as it provided information about the areas 
most in need of action to bring about immediate improvement. 

c) Multi-output model. We believed it was necessary to give our model 
sufficient flexibility in order for it to be used at different levels of management. The 
specific formula of decentralization that operates within the University obliged us to 
create a mechanism that, through partial use, would be able to offer indicators of 
quality and information about improvements that could be adjusted to the different 
levels of decision-making within the University hierarchy. 

The model uses the primary information derived from the questionnaire modules 
and considers the importance that should be given to each of these through the introduction 
of two blocks of coefficients. 
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a. Those that result from calculating the aggregate of student opinions about 
the importance that should be given to each topic in order to obtain a quality 
service, and 

b. Those obtained by introducing into the operational mechanism the factor 
“number of students” affected by the scenario which the indicator refers to, in each 
case. 

A simple frequency analysis of the variables that represent the primary information 
offers data as to the areas of the service that require improvement. Moreover, the mix of 
variables and weightings, simplifies the process of calculating the service quality indicators 
adapted to the responsibilities of one specific lecturer, a group of lecturers (e.g. those 
teaching a particular subject), a department, a degree program, a centre, or even the entire 
University as an institution. 

2.1. Data collection and analysis 

To test the reliability and validity of the self-evaluation protocol, we carried out a 
pilot study that was limited in scope to students registered in the University of Seville 
School of Business Studies during academic year 1996-1997. Specifically, the fieldwork 
took place during May 1997, when we collected a total of 23,585 questionnaires, given that 
each module of the complete protocol was considered a separate questionnaire, with a total 
of 359,000 data items. The research brought us into contact with some 30% of the student 
population registered in the centre during the year in question (1,560 students from a 
population of 5,234). The scope of the sample involved was such that the results could be 
extrapolated to the total population with a margin of error in the order of ±2%, and a 
confidence level of 99.73%, always taking into account the variable with the greatest 
degree of variance in the sample. 

Prior to statistical analysis, the data was revised to ensure that the basic criteria on 
which it was to be judged could be met. Finally, data was recorded in a format compatible 
with the SPSS statistical software program and was then subjected to six analyses: 

1) We projected the frequency distribution for each of the variables and 
calculated the measures of dispersion and central tendency. 

2) Cronbach’s a was calculated in order to estimate the reliability of each of the 
modules that made up the full self-evaluation protocol. 

3) Factor analysis of the sample data (module by module) was carried out in 
order to establish whether or not simplified mental schema existed in the minds of 
the interviewees. This was preceded by statistical validation to ensure that it was 
appropriate to apply factor analysis to our data. Similarly, Cronbach’s a was 
calculated for each of the factors we identified in order to establish their internal 
consistency. 
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4) Multiple regression was used to test the impact that each question had on the 
overall value of the “Global Satisfaction” (GS) variable included within each of the 
modules. 

5) Correlations between GS and the simple aggregation of the other items in 
each module were calculated. Thus we ascertained the internal validity of the 
measures by establishing whether or not they did effectively refer to the construct 
we were studying. 

6) Analysis of crossed tabulation of the various GS variables and those that 
represent the profile of the interviewee. By applying the test and calculating the 
proportional reduction of the standard error of estimate, we aimed to verify the 
extent to which student satisfacion could be associated to elements that are external 
to the actual service offered and therefore beyond the remit of decision-makers 
within the University administration. 

3. RESULTS 

Frequency analysis enabled us to learn from many different angles (that of each 




Figure 1. Students’ perceptions of their level of satisfaction. 



lecturer, each subject, each centre) the aspects of university services that, in the opinion of 
the interviewees were most in need of urgent reform. Similarly, we were able to establish 
the relative values with which (in the case of the University of Seville’s School of Business 
Studies) to weight the levels of importance to be given to each area with regard to the 
construction of the corresponding indicators of student satisfacion. In this respect, the 
interviewees indicated unambiguously the different degrees of importance they attached to 
lecturers’ teaching performance in their perception of their level of satisfaction with the 
service they received (see Figure 1). 

The a coefficients were systematically higher and closer to the optimal value (1), 
which demonstrates the reliability of the instrument (see Table 1). 
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Table 1. Alpha coefficients for all modules. 



MODULE 


COEFFICIENT a 


TP 


0,96 


SPD 


0,83 


CD 


0,79 


ssc 


0,68 



Factor analysis offered information about the underlying structure of the data 
provided by the sample. By using the principal component method (with eigenvalues 
greater than 1). By varimx rotation, we identified in every case the factors of clusters of 
variables (though the potential number of these had not been specified in advance). Of 
special interest are the results obtained in relation to the TP and SPD modules, where there 
were significant clusters of items, to the point that only two factors could be distinguished 
in each case. 

All of the questionnaire modules proved useful in constructing the indicators 
representative of the global level of student satisfacion. The correlations between the 
variables “Questionnaire total” and the corresponding GS, were sufficiently high, positive 
and close to the optimal value of 1 (see Table 2). 

Table 2. Correlation coefficients calculated from the sample data. 



MODULE 


COEFFICIENT p 


TP 


0,89 


SPD 


0,68 


CD 


0,58 


scs 


0,64 



We should not be deceived by the gradual reduction in the values of these 
correlations. Our evaluation model takes into account the possibility of just such an 
apparent loss of validity. This is resolved by including the information used to calculate the 
lower levels in the calculation of the indicators of the higher ones. 

The multiple regression studies led us to identify the variables that most influenced 
global student satisfaction. The adjustment made in the TP module is of special importance 
here, as the coefficient of determination (R^) was 0.82 (maximum = 1). 

Finally, by contrasting data from the student profile descriptors and student 
statements of satisfaction, we obtained evidence of the existence of certain associations. 
These were indicated by calculating Goodman and Kruskal’s X and x, which show the 
proportional reduction produced by the estimate of error whichever dependent variable was 
used. If we take the GS variable as such, the tests we carried out seem to show that prior 
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knowledge of certain characteristics of the student profile will lead to a considerable 
reduction in the level of error if we want to predict the level of satisfaction students will 
express. 

3.1. Conclusions 

Based on these results, we have reached six significant conclusions. 

1 . Our questionnaire has shown itself sufficiently reliable as a research tool in 
all respects. It is flexible enough to be fine-tuned in order to adapt it to the particular 
requirements for information that any specific user might have. Here, we should bear in 
mind the fact that the term reliability can be interpreted in two complementary manners; 
“test- retest” reliability - that is, when repeated use of the same instrument, in objectively 
similar contexts, leads us to obtain very similar results in all instances; and, reliability in the 
sense of “representativity” or “completeness”: if we asume that exhaustive research into 
any social reality implies considering an infinite number of items, we should be reasonably 
confident that the items we choose are a suitable, representative selection of those possible. 

2. Factor analysis indicated that students followed very simple thought schema 
in order to decide on their statements of satisfaction. If we look at the TP module, the study 
shows that two major factors exist. The first encapsulates all of those items that are in some 
way linked to what we can call the apparent professional teaching skills of the lecturer 
(62.9% of variance). The second factor incorporates all of the variables related to the 
lecturer’s apparent ability to generate a positive climate in terms of human relationships 
(7.4% of the variance). 

3. Two major factors also appear in the SPD module, and together they account 
for 52.5% of the variance. The first includes all the variables related directly or indirectly 
with the systems of evaluation used in the subject. (1 1.3%). By a process of elimination, all 
of the remaining aspects of programming (those not connected with evaluation) are bound 
up in a second factor, which accounts for 41.2% of variance. 

4. Our research tool instrument demonstrated itself internally valid in achieving 
our objective of measuring different levels of student expressions of satisfaction with the 
services offered. 

5. Some variables had a greater influence than others when establishing the 
level of student satisfaction. If we limit ourselves to the TP module, the clarity of the 
lecturers’ explanations and the extent to which they are able to lighten these and make them 
accessible to students, are most significant determinants of the overall level. Some way 
behind these, in terms of their influence on the GS variable, come issues such as how well 
prepared the classes are, a concern to resolve difficulties student might have, or the effort 
made to stimulate the students’ level of motivation. Curiously, questions such as the extent 
to which the lecturer faithfully follows the subject program, or the effort made to encourage 
participation, are barely worth mentioning in this context. 
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6. Statistics for analysis of crossed tabulation and proportional reduction of 
estimate of error suggest that certain characteristics of students’ personal profiles are 
closely linked to their expression of satisfaction. Specifically, the sample data show strong 
associations between the GS variable and the degree to which students commit themselves 
to following the subject (measured in terms of attendance at class, frequency with which 
they attend tutorials, and day-to-day attention to the subject, among other things). Those 
who demonstrated greater involvement were consistently more satisfied. In contrast, GS 
was inversely associated with student age, and with the number of years they had been 
studying at the University. Older students, or those with longer experience of University 
life, if we can express it like that, were also those who showed greater levels of 
dissatisfacion with the characteristics of the centre where they were studying. Although we 
cannot really draw definitive conclusions as to the cause, it does seem possible to affirm 
our belief that at least part of student satisfaction is due to factors other than those involved 
in the services provided by the University. 
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